Skip to content

Add Gemma 4, Ministral 3 and Aya Vision through one chat-template wrapper - #2

Closed
SalimAlkharsa wants to merge 16 commits into
eval/protocol-parityfrom
eval/new-models
Closed

SalimAlkharsa wants to merge 16 commits into
eval/protocol-parityfrom
eval/new-models

Conversation

@SalimAlkharsa

Copy link
Copy Markdown
Collaborator

Summary

Adds five new vision-language models through one generic wrapper, models/hf_chat_vlm.py:

Key Checkpoint
gemma4-e4b google/gemma-4-E4B-it
gemma4-12b google/gemma-4-12B-it
ministral3-8b mistralai/Ministral-3-8B-Instruct-2512-BF16
ministral3-14b mistralai/Ministral-3-14B-Instruct-2512-BF16
aya-vision-8b CohereLabs/aya-vision-8b
  • The wrapper sends one image + text turn through the processor's chat template and greedy-decodes only the new tokens. Thinking is off by default (enable_thinking=False). Any reasoning block is cut at the token level, because Gemma 4's <|channel>/<channel|> markers are special tokens.
  • Registry rows use MIRAGE_<KEY>_MODEL_PATH overrides. --model choices now come from the registry.
  • requirements.txt moves to transformers==5.19.0 (the release that loads all four families; Gemma 4 12B needs ≥5.10) and adds mistral-common==1.12.0. The PaliGemma and Qwen2-VL wrappers still import under tf 5. Old-model results are replayed from cache, not regenerated.

Validation

  • pytest -q on the Grace login node: 23 passed.
  • Smoke on Grace A100 40 GB (bf16, greedy, tf 5.19.0). Each model loads and answers POPE/ROPE through the chat template. Peak GPU memory:
Key Snapshot Peak GPU memory
gemma4-e4b ee0ef602 16.0 GB
ministral3-8b f6fae979 18.2 GB
aya-vision-8b d7601ac8 18.6 GB
gemma4-12b 707f0a3b 24.1 GB
ministral3-14b 3cea74c1 28.6 GB
  • POPE answers are plain yes/no. ROPE answers are full sentences ("The object inside box 2 is a black pot…"), so ROPE strict accuracy is low for these chat models, as it is for Qwen2-VL. The prompt and metric are unchanged.

…pper

models/hf_chat_vlm.py loads any processor/auto-class pair whose chat
template takes one image and one text turn, decodes only new tokens,
strips reasoning blocks and keeps thinking off by default. Register
gemma4-e4b, gemma4-12b, ministral3-8b, ministral3-14b and aya-vision-8b
with MIRAGE_<KEY>_MODEL_PATH overrides.
@SalimAlkharsa

Copy link
Copy Markdown
Collaborator Author

This PR looks good and can be merged but want to merge in order w everything reviewed

# Conflicts:
#	run.py
The generic chat wrapper keeps its body; the registry points at the new
module path.
No driver called it: every published cache row came from single-item
Predictor.predict, and padded batches can differ from single-item
outputs while sharing their cache keys. Removed BaseModel.batch_generate
and batch_inputs, use_true_batching, the PaliGemma and Qwen2-VL
_batch_generate_parallel/_prepare_batch_inputs, HFChatVLM.batch_generate,
Predictor.batch_predict and its alias, and both backends'
batch_generate. The THRONE evaluator keeps its own batching (it calls
Predictor.predict). generate() is unchanged.
get_model_info (and the PaliGemma/Qwen2-VL overrides), update_config
and the context manager had no callers. base.py imports torch only
inside the bitsandbytes branch, and models/__init__ resolves classes on
first use, so importing models needs no torch. get_quantization_config
stays: both load_model methods call it.
BaseModel.cache_name is the model name that goes into prediction cache
keys; LocalModelBackend uses it and falls back to the class name for
objects without one. PaligemmaModel, Qwen2VLModel and HFChatVLM pin the
names their caches were written under, so splitting or renaming the
wrapper files cannot change a key.
gemma4.py, ministral3.py and aya_vision.py hold thin HFChatVLM
subclasses that inherit its cache name, so their keys do not change.
predict.factory.REGISTRY (read-only) maps each key to module:Class, its
model path variable, default HF id, and mirrors of the cache name and
dtype; identity_for(key) computes a model's cache identity from those
without importing torch. qwen2vl-2b and qwen2vl-7b get their own rows
(MIRAGE_QWEN2VL_{2B,7B}_MODEL_PATH, falling back to
MIRAGE_QWEN_MODEL_PATH) so results and verdicts are keyed by size;
qwen2vl stays as it was. Every other key, variable and id is unchanged.
run.py and demo.py use the public name.
build_predictor(model, key=...) wraps an already constructed model (for
tests or a model outside the registry) at the same entry point the CLI
uses; key is required because it names the model in results and
verdicts. A string or None still selects a registry model as before.
cache_only=True opens the default cache read-only.
build_model checks the registry cache_name/dtype mirror against the class,
a shared Qwen fallback path must match the row's checkpoint, and
build_predictor rejects a model key that disagrees with key=. run.py uses
build_predictor(cache_only=...) and builds the FaithScore verifier from the
registry. Drop the unused ModelNotLoadedError imports.
@SalimAlkharsa

Copy link
Copy Markdown
Collaborator Author

Noted, the stack will merge in order, #1 → #6. Since this comment, #2 has been refactored: one HFChatVLM base in models/hf_chat.py with a thin file per model family, and the model registry in predict/factory.py. Worth one more look; the head is now a859e45.

Comment thread predict/factory.py
_BF16 = "torch.bfloat16"
_QWEN = "MIRAGE_QWEN_MODEL_PATH"

REGISTRY: Mapping[str, Entry] = MappingProxyType({

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do changes in this mapping require to be done for any new model? If so makes sense to add a check saying id added model must update this part

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. A new model needs one row in REGISTRY, plus a _MODULES line in models/__init__.py if it's a new class. It's enforced already: an unknown key raises, and build_model raises if the class's cache_name or dtype doesn't match its row. As of be25b9f, those errors say "add a row to / update the row in predict.factory.REGISTRY", and the registry has a one-line header comment saying every model needs a row.

@SalimAlkharsa
SalimAlkharsa deleted the eval/new-models branch October 19, 2026 04:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant